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. I IMTRODUCTION 



A. PURPOSE 

Except in instructional situations where student programming is the 
primary, a^^^ microcomputers can be used to assist the instructional - 

process effectively only to the extent that quality software is available. 
This Evaluator's Guide and the associated evaluation forms have been 
designed to assist teachers in identifying such software. 

The development of the guide and the evaluation instruments have been- 
based on the philosophical approach that description and evaluation of 
courseware should be separated. This has been done in the three major 
sections of the instrument - CONTENT, INSTRUCTIONAL DESIGN AND TECHNICAL 
DESIGN. - ^ 

B. PROCESS \. 

The process of evaluation conducted by Alberta Education involves the \ 
Program Development Division ^nd the Computer Technology Project's 
Clearinghouse. As priorities are identified by the Program Development • 
Division, the Clearinghouse solicits courseware from publishers for 
evjaluation. It then has the materials described and assessed by 
teachers who have, the appropriate knowledge in the subject are^^^^ 
consideration and who have received trainir^ in the use of the evaluation 
instrument. 

These evaluations result in one of three decisions: 

a. The material is identified, as worthy of further evaluation. 

b. The luaterial is identified as having limited use in the classroom 
(Supplementary) . 

c. The material is rejected. 

Courseware identified in "a" is evaluated further by . the Program Develop- 
ment Division for congruency to the program of study and may~ be designated 
as a* "Prescribed**, "Recommended" or "Supplementary" learning resource 
for Alberta schools. Prescribed and Recommended materials are acquired 
and distributed through^ an. agency identified by Alberta Education. All 
courseware "evaluations are filed in the Clearinghouse and results are 
made available to Alberta schools. - 
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C. DESCRIPTION OF THE IMSTRim 

Courseware currently being published may come in a number of configurations. 

■ ''-^ 

a. A disk may contain a number of ''stand-alone" programs. For 
example, programs on spelling, identification of capital 
cities and review of- basic number facts may be available 
all oh one disk. ' ' , 

b. A disk may contain a number of ^'programs tied together under 
the same basic goals but each having different objectives. 
For example, a disk may have four programs *- addition.,, 
subtraction, multiplication and division of decimals. 

c. A disk may contain one main program, for example, a 
simulation of a sciontific experiments 

d. Two or more disks may be tied together into a package or 
series. For example, six disks reviewing arithmetic 

f undamentals^ may be combined together into a single package 
with a management system. v 

As the basic item of purchase is the disk, IT Jllv-THE DISK which has 
been made the basis of evaluation of this itistrument. 

There are t^o parts to the evaluation instrument - the PROGRAM 
EVALUATION FORM and the DISK EVALUAT I ON FORM . 

A PROGRAM EVALUATION FORM is generally completed for each stand-alone 
program. on the disk. The form has descriptions and evaluations of a 
program's intents, (Content, instructional design and technical design. 
Each stand-alone program on the disk receives an evaluation that is 
independent of other programs on the disk. ' ^ v 

A disk containing a number of programs tied together under a common 
goal will generally have each of the prbgrams evaluated separately. 
However, when -the programs are designed in such a manner that the 
only noticeable difference is the content and all other factors of 
instructional design and technical design are identical, one Program 
Evaluation Form may be used for all the programs. 

A DISK EVALUATION FORM is used to evaluate the overall DISK . It 

contains the identification of the ,disk (title, publisher, etc.), ,^ 

and a description and evaluation of any management system that may 

exist. It also contains a summatiye evaluation of the disk's entire 

contents, instructional design and technical design. A summational 

statement and a recommendation are at the. end of the form. ' 



I 
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II PROGRAM EVALUATION FORM 



A. INTENTS 



OBJECTIVES List the objectives /that the program is intended to 

achieve and indicate if they are stated oir. inferred. 

Objectives are the behaviours that' the learner manifests 
as a result of involvement with, the materials. The 

evaluatbr shdulci ask himself what it is the learner 

will be able to do after using these materials? 

Examples of objectives: Learn^ers will be able to add 
fractions with common denominators; learners will be 
able to recognize the names of the capital cities of 
Canada; learners will be able to identify the correct 
use of "their, there and they're"; liearners will be 
able to identify the chemical elements given the ^> 
abbreviations of the elements. 



B. CONTENT 



PROGRAM 
CONTENTS 



Describe the educational contents of the program 
(topics, skills, concepts)!, . 



1. What are the topics/ skills/concepts covered in 
the program? (range) 

2. What is the progression of the presentation? 
Hqw are the\units ordered? (sequence) / _ . 

3. How intensively is each dealt with? What does 
the treatment include? (explanation, rules, 
examples , illustrations , drills , tests , etc . ) 



Example: This program covers three types of addition 
of fractions. Students begin with addition of 'common 
fractions with 'the same denominators, proceed to 
addition of fractions where one fraction has a denom- 
inator that is a factor of the other (2 and 6, 3 and 
12) and finish the program, with addition of fractions 
with completely different/denominators (3 and 4, 5" 
and 7) . Improper \and mixed fractions are not included 
in the program. The presentation of each topic includes 
one page of introduction, t^o examples and a brief 
drill, A statement of rules is available. The program 
ends with a test on all three skills,' 
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Example: This program covers three types of improper 
word usage: 



there/their/they 're 2 . too/to/ two 



good/well 



Each example is presented through the use cff a rule, 
one example of correct usage and one example of 
incorrect usage. A drill of all three types is 
included at the end of the programs. <, 



USER 

SUPPORT 

MATERIALS 



User-support refers to STUDENT support materials. 

1. Describe the extent of any student materials 
Such as ,pre-instructibn activities, work sheets 
and follow up activities that are included with 
the disk. 

L 

2. Indicate if the materials are integral- to the 
design of the package or if they are optional. 



CONTENT 
EVALUATION 



Range 



Write a narrative statement which will evaluate the 
content described earlier. • Some questions to be 
addressed include: 



IS the range of the content appropriate for the 
target audience? 



Is the range of the content effective, in achieving 
the intents of the program, is sufficient content 
covered? is too much covered? 



Sequence 



IS tfie sequence of presentation appropriate for , 
the target audience? Is the size of the increment 
appropriate? 



Is the sequence effective in achieving the intents 
of the program? 



/ 
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Depth 5. Is the depth appropriate for the target audience? 

G; Is the depth effective in achieving the intents 
of the program? 

/• ■ . ^ • ' 

- Is the material presented in sufficient depth 

and intensity so that learning will take place? 

• ~ Are definitions and explanations available where 
necessary? 

- Are sufficient examples and illustrations used? 

/ 

Is sufficient practice available? 

Accuracy, 7. Is the content accurate? Does it contain factual 

errors, an invalid -model in a simulation, an 
^ oversimplified model or series of examples, in- 

accurate graphs or displays? 

8. Is the content current? 

Clarity 9. Is the content presented in a clear and concise 

manner which is appropriate for the target audience? 

Bias 10. Does the content include biased information? 

;.' ■ / 

- e.g. improper language usage? 

. - e.g. ethnic or social class references? / 

- e .g . references to violence? / 

- e.g, references to sexual or social roles? 

- e.g, age portrayals? 

~ Does it contain Canadian data where relevant? 

- Is content metricated? 

Curriculum 11. Is the content appropriate to the Alberta curriculum? 



User. Support ^2. 
Materials 



Are user support materials appropriate for the 
target audience? 



13. Are user Support materials effective? 



- Do they contribute to the learning process? 

- Are these materials sufficient? 



/ 
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C. INSTRUCTIOMAL DESIGN 



INSTRUCTIONAL 
TECIIIIIOUC: 



Check the appropriate box(es) . h definition of 
terms is included at- the end of the guidebook. 



STUDENT 
INTERACTION 



Describe the student interaction that is required 
in the program. 



1. How much student interaction is there? 

2. What type of student interaction is there? 
(multiple choice entry, input of answer, "press 
return", etc.) ■ . ^ , 

Example: Student interaction in this program 
consists of "press space bar" instructions at 
the bottom of each page (approximately 25) and 
the input of the number of the correct response 
(1-4) in the ''.rill section. (5 questions) 

Example: Student interaction in this program / 
.consists pf selection in the menu, page turniTig 
devices and input of answer in the drill sec4:ion. 
. In the menu, the student is asked. to enter he 
name of the program section desired. During 
the explanation, (10 pages), the student can go 
forward or backwards by using the right or left 
arrow keys. The drill c[aestions require the 
student to enter the correct answer along the 
% sign. ' 



FEEDBACK 
TECHNIQUES 



Describe the feedback techniques, 
that may be answered are: ^ 



Some questions 



1. When is feedback provided? (after all user 
responses, when user response is wrong, randomly 
on correct responses, etg . ) 

( ■ ■ , 

2. 'What type of feedback is-'provided? (e.g. graphics, 
. text , audio , combinations;,,/ etc . ) 



3. • What content is included in the feedback? (e.g. 

general message, specific hints depending upon 
the error, the correct answer, a restatement of 
the correct answer, explanation of incorrect 
response, etc . ) ; ' 

i! . ■ ' ^ ■ 

4. What IS the extent* of the variety of the feedback? 
(e.g. the, same phrase "try again" is used after 

' each incori^ecit response) 
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Is quantitative feedback provided? 
9 questions correct out of 20) 



(You got 



Example: There' are two types of feedback 
employed. On inputting a correct answer, the 
student receives a brief congratulatory message 
chosen randomly from approximately 10 statements. 
If an incorrect response is entered, the student 
is first told to "try again". If, on the second 
attempt, he is still wrong the correct answer is 
given.. Quantitative feedback is available at the 
end of the drill. 



EVALUAT I ON 
TECHNIQUES 



Example: Graphical and andio feedback are employed 
throughout the program. Correct answers are 
rewarded with a complicated graphical picture 
(4 pictures available) . Incorrect answers are . 
punished by a large X which flashes on and off 
the screen. At the same time, a loud buzzing 
noise advises the user ^hat a mistake has been' 
make . i 



Describe the evaluation techniques that are used to 
determine the student's success in the program? 

1. Is there a pre-test or a post* test? 

/ 

2. What are the criteria for success? Who sets the 
criteria? 

Note that within this context, x a drill is not a test. 
A drill presents questions, often in a structured way, 
to give the user practice in a particular skill. 
Often, more than. one attempt at a question is given 
and hints may also be provided. 

A test determines if the practice of the skill, the 
""nvolvement of the student in the program, has been 
or is likely to be successful. Questions are not 
structured to lead a student towards. more difficult, 
'questions, only one chance is given and feedback is 
limitejd to an indication of right or wrong. 



BRANCHES 



Describe the branches that exist within the program. 
What types of branches are. available to the user? 
Is" the program linearly sequenced or is there pro- 
vision for review, remediation, enrichment, help,, etc.? 
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Describe the type of control that is used with each 
of the following: Sequence, Rate and Amount. 

There are three types of control - PROGR?VM, TEACHER 
and USER. Program control is based on pre-set criteria 
built 'into the program while teacher and user controls 
'are based on entries „inade during the operation of the 
program by either the teacher or the'^student * (user) . 

Sequence refers to the order of the presentation, the 
path of the user through the branches described above. 
Under what circumstances does the user enter these 
branches? For example: Sending a student to a remedial 
loop automatically if he makes a certain number of 
errors is program control . User control gives the 
student the opportunity to go to a particular part of 
the program, e.g. skip the explanation and go straight 
to the drill . I Teacher control allows the teacher to 
specify, before a student begins the' program, what 
parts of the program he is to do. 



Rate- refers to^ the pace of the presentation, the time 
allowed to read each display and to answer questions. 

Amount . refers to the amount of material that is pre- 
sented in any one branch and how it is controlled. 
For example, the\ number of questions done, vthe niimber 
of examples seen; the number of pages rea[d or re-readi 

\ . ' ' ' - 

Example: . \ 

Sequence-User: \ The student ma:y elect to skip the 
instructions and go directly to the drill. 

Rate-User: The \pace is controlled by pressing the 
space bar. \ . I 

\Amount-Progr^ : All students do 20 questions in the 
drill. If the instruction section is^/chosen, all •> 
students see the same number of examples and. the same 
number of pages. \ I 



\ 
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INSTRUCTIONAL 
DESIGN ' 
EVALUATION. 



/ ... ,•• ■ ■ 

Wri-te a narrative statement which will evaluate the 

instifuctional designs described earlier. Some 

questions to be addressed include: 



Instructional 
Technique 



/ 

Student ' 
Intera^ition 



Feedback 
Techniques 



5. 
6. 



IS the instructional techniqufe appropriate for; 

the target audience? ' 

■ I 

" • ' / 

IS the instructional technique effective? 

Is it' appropriate for the intents of the program 

and the composition of the content?. 

Is the student* interactior^ appropriate for the 
target audience?^ ^ 

- Is the -type of interactroh appropriate?. 

- Does the time required to complete the program 
exceed the attention span of the intended users? 

\ . ■ - . • . ^ 

Ife the student interaction effective? 

• i. - 

- Is there sufficient interaction to meet the 
intents of the program?^ Does the student 
participate or just watch? ' _ 

- Is a consistent type of interaction used? 

Is the feedback appropriate to the target audience? 
Is the feedback effective? . . / 



Are user inputs accurately evaluated as right 
or wrong? ^. 

Does the, feedback distinguish between a wrong fi 

answer arid the wrong 'format? 

Is the feedback non-threatening immediate 

and positive? \ 

Is the feedback corrective when ne^ssary? 

Is the feedback relevant Nto a student's responses? 

Are traps set toVcatch potential errors of any 



kind? , . . 

Does the feedback explain why a response is wrong? 
Are cues or prompts used after a wrong response? 
Is the feedback to a wrong response unnecessarily 
attractive? "\ 
Can the user or the teacher control the feedback 
(e.g. type, amount when given)? 
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Evaluation 
Techniques 

■ 1* 

1 



7. Is the evaluation tebhnique appropriate to 
the target audience? Are the criteria for - 
success appropriate. for the ability/skills 
of the intended user? 

8. Are the evaluation techniques effective? 

- Is the absence/presence of a post test 
appropriate to the intents of the program? 

- If present, does the test measure the 
user's achievements on all the intents? 

- If present, does the test content accurately 
reflect the content of the material presented? 



Branches 



9. Are the branches appropriate to the target 
audience? 



10. Are the branches effective in accommodating 
individual differences? Shoui|-d there be 
more? less? 



Types of 
Control 



11. Are the types of control appropriate to the 
target, audience andT' are they within their 
capabilities? \ 

- siBquence,? x \. 

- rate? 

- amount? 



12. Are the types of control effective? Does 
the user have control of the program when 
necessary? Can the teacher control the 
program when necessary? 
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D. TECHNICAL DESIGN 



COLOUR, SOUND 
GRAPHICS ' . 



Describe the extent to which colour, sound 
and graphics, are used. (Indicate if the 
evaluation was done without a color monitor) 



TECHNICAL DESIGN 
EVALUATION / 



Displays 



Write^ a narrative statement which will evaluate 
the technical quality of the progranr: Some - 
questions to be addressed include : 

1. Are displays appropriate for the target 
audience? ; _ 

- Are vocabulary, sentence structure, sentence 
length, character size and general readability 
appropriate? 

2. Are displays effective? 

- Are they free from grammar, spelling, punc- 
tuation and hyphenation errors? 

- Is the amount of material presented at one 
time appropriate? 

( - Is the material clear and easy to read? Is 
r text material too lengthy or wordy? Is 
there adequate spacing for clarity? 

- Do message design iprinciples place emphasis 
where the important concepts are? Is 
scrolling used appropriately so that pertinent 
information is retained on the screen? 

- Is .the transition from display to display ' 
smooth, well paced and unobtrusive? Is 
the user given too little/ too much time to 
read and^absorb the information given in 
each display? ' 



12 



ALBERTA EDUCATION 



1 



PROGRAM NAME(S) 



INTENTS 

OBJECTIVES ; 



PROGRAM EVALUATION. FORM 



Version 1 



Stated 



Inferred 



Describe the objectives 
(Guidebook, page 3) 



CONTENT DESCRIPTION 

Describe the program contents: 



(Guidebook, page 3) 



Describe user support materiials (if any) : (Guidebook, p^ge 4) 



CONTENT EVALUATION (Guidebook, page 4) 



y 
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NSTRIICTIONAL DESIGN DESCRIPTION . 

Check the appropriate instructional technique (s) used and comment if necessary: 

Drill and Practice Tutorial "Simulation Game 

Other (specify) x 

Describe the student interaction in the program: (Guidebook, page 6) 



Describe the feedback techniques used in the program: (Guidebook, page 6) 



Describe. the evaluation techniques used in the program: (Guidebook, page 70 



/ 



Describe the branches available in the program: (Guidebook, page 7) / 
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ALBERTA EDUCAT.ION 



VERSrON 1.2 



OESCRIPTION 



Disk Title: - - 

Series Title: (if ;%licable)^ _ 

Producer : . f- '"'' ' ■ ' -A 
Address: ■ 



1 



Telephone No.:_ 
Subject Area:_ 
Topic (s) . 



Specified Users :^ 



•Required, Hardware and Software: . 
(in addition to an Apple II, 48K, Applesoft, 
Disk Drive, DOS 3.3, B&W Monitor) 



Version: 
Gost:,, 



Programs Included in this Evaluation^ 



i' 



Other Components Included in Package: 



Available in Languages other than English? 
No __ Yes - (specify) . 



. MANAGEHENT SYSTEM DESCRIPTION (Guidebook, page 14) 

Is a .management system available on this disk? ^ 
Describe the functions available: .(if any) 



on' another disk? 
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.. EASE OF USE 

. Describe any special prerequisites:. 

How difficult/easy is it for a teacher to use the disk? 



DISK CONTENT EVALUATION (Guidebook, page 4) 




DISK INSTRUCTIONAL DESIGN EVALUATION (Guidebook, page 9) 
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DISK EVALUATION; SUMMATIONAL STATEHENT; MAJOR STRENGTHS AND WEAKNESSES 



RECOHMENDATION 

I recommend that 



this disk be evaluated further to determine , the full\iature , 
of its use. (Prescribed or RecoMended) ' 
this disk be considered for supplementary use in the classrooml 
this disk be rejected. ■ • 



I agree that the copywrite to this report rests with Alberta Education and confirni that 
no attempt was made to copy; or modify the program or the accompanying documentation. 



Check here if observations 
of student use were 
incorporated into this 
evaluation ■ . 



Name of evaluator: 



Signature: 



FRir 
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Describe the capacity: 
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Describe the types of control in the program: (Sequence, Rate, Amount) 

(Guidebook, page 8) 




TECHNICAL DESIGN DESCRIPTION / , i n 

Describe the use of colour, graphics and sound in the progr^: (Guidebook, page 



TECHNICAL DESIGN EVALUATION (Guidebook,, page 11) 



ADDITIONAL COMMENTS 
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Colour 

Graphics 

•Sound 



3. 



Are colour, graphics and sound appropriate 
for the target audience? , 



4. Are colour, graphics and sound used effectively 

- Are they. used appropriately for the content 
material? r . 

- Is there an option for the use/non-use of 
sound? Will the sound distract others in 
the classroom?. 

- Are they appropriately mixed with text so 
as to 'give variety to the presentation? . 

- Are the graphics boring? repetitive? 
too slow? ' 

- If colour is present, will the program be 
effective on a non-colour monitor? 



Ease of Use 



Can the intended user easily and independently 
operate the program? • , 

- Does the operation of the program interfere 
with the user's concentration? 

- Is impossible to get lost in the program* 
with no way out? . , 

~ Are instructions available at likely points 
and are they appropriate for the target 
audience?' 

- Can the user exit the program, return to 
the menu or move to another section easily^ 
when necessary? • • 

- Is the program reliable and bomb-proof 
under normal use? 



Ill DISK EVALUATION FOR^ 



A. DESCRIPTION 



This section will be completed by the Clearinghouse. Some of the 
sections are described below. 



Series Title 

Version 
Cost 



ablisher 
/ idress 
Telephone 



Specified Users 



Required . 
Hardware & 
Software 

Programs 
Included 



Other 

Components 



If the^ disk being evaluated is part of a series,- 
the name of the series will be entered here 
(e.g. MilTiken Math Sequences). 

Version number or date of publication. 

The most recent cost as provided by the publisher. 
This' is subject to change without notice by the 
publisher. 

The original source who produced the package. 
A telephone number is provided (if possible) 
so that schools may contact the publisher in 
order to determine if there is a Canadian 
distributor . 

Grade level, grade range or other ability level 
indicators for which the package has been 
specifically intended . 

Hardware or software beyond the standard package 
of 48K, 1 disk drive, 3.3 DOS, B&W monitor. 



A list of titles of each program on the disk. 
One Program Evaluation Form will generally, be- 
completed for eac^h title. 

Other component^which are included in the package 
such as a management system, student .^workbooks or 
a teacher guide. 
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B. mNAGEMENT SYSTEM 



V 



FUNCTIONS 
DESCRIPTION 



Describe the functions available in the management 
system. . These may include: 

Clas3 lists: e.g. initiating., maintaining^ adding 
to, deleting from; \ 

Reports :« e.g. by individual student, by class, 
by specific unit; 

^ ... .A . . ' 

Prescription: e.g . ^ prescription of specific 
assignments or a range of assignments for individuals 
or for the class; 

Diagnosis: diagnosis of a student's weaknesses; and 
Security: passwords, etc. 



CAPACITY 



How many students can be kept in one class? How 
many scores can be kept for one student? Hew 
many classes can be, kept? 



EVALUATION 



Write a narrative statement which will eValuate 
the effectiveness acid the ease of use of the 
management system. Some c[uestions to be addressed 
include: 

1. Do the functions work as described? 

2. vis the system effective in meeting the needs 

of the tieacher? 

I ' ■ '. ■ 

3'. How easy is the system to set up and use?. 

4. Is the capacity sufficient? 

5. Is data easily .and effectively collected? 
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C. ./EASE OF USE 



PREREQUISITES 
TEACHER USE 



Describe any special prerequisites that are 
required beyond the normal grade level skills. 

Evaluate how difficult/easy it is for a teacher 
to use the package- Consider the ambunt of dpc~ 
umentation and support that comes with the package 
both inside and outside of the program. Some 
questions to be considered are: 

1. Can a teacher control the pace/difficulty if 
desired. ■ - 



5. 



Do the students require a minimum amount of. 
teacher supervision when using the program? 

Are error handling and identification proce- 
dures sufficiently detailed so that a teacher 
can help. a student^ 

Is the disk adaptable to a variety df class- 
room learning environments? 

Does the disk require a minimal amount of 
equipment manipulation? 

Can the disk be used by a teacher with a 
minimum' of computer competency? 



D. DISK EVALUATION - . ^ . ' 

Evaluative statements in this section cover, the. entire disk. If there 
is more than one program on the disk and if two or more Program Eval- 
uation Forms have been completed, this section will include siammaries 
of the evaluative coiranents made on those forms. If only one Program' 
Evaluation Form has been completed, the reader should be referred to 
evaluation statements made^on^ that form. 

. ' ■ ^ ' ,. ■ , " ■ ' /: . ■ . ■ 

Disk Content-- see F-g^ four of this guidebook 
Disk Instructional Design - see page nine of this guidebook 
Disk Technical Design - -see page, eleven of t^his guidebook: 



E. SUTflATION 

EVALUATION 
SUMMARY • 



RECOMMENDATION 



Summarize the strengths and weaknesses of the disk. 
Considering the comments made in the sections above 
what is your evaluation of the entire disk?^ " 

Check one recommendation . See the guidelines for:: 
j^i .making recoipnendations on page sixteen and the ' 
''necessary features of tutorial/drill programs on ^ i 
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* • IV GUIDELINES FOR MAKiNG REOTENDATIONS 

There are no hard and fast rules with which to equate the strengths 
and weaknesses of a disk to "the correct recommendation". Recommend- 
ations are subjective and will vary from one evaluator to another. 
We hope that the degree of variance will be small and that the process 
of forming a synthesis from three evaluations will minimize any dif- 
ferences • that might occur. 

However, to keep these variances small and. to strive for consistency 
"across subject and grade levels, . it- would be advantageous, to consider 
some . guidelines. — 

Within this instrument, there are two questions which have been addressed 
in almost every evaluation section: 1. the appropriateness to the target 
audience, and 2. the effectiveness in meeting the intents of the program 
;A program that. is ineffective should be rejected. ' On the other hand, 
a program which is inappropriate to its target audience may still be 
effective for another target audience and could be considered for 
limited use • (supplementary ) in the classroom. 

What makes a program ineffective or inappropriate? Certainly the 
degree of the weakness has to be considered. A program should not be 
rejected because, of a minor flaw. The number of weaknesses should 
also be considered and a program with 4 or 5 minor flaws may have 
its use restricted because of the cumulative effects of small weaknesses. 

Major flaws are those which_ seriously hinder the learning that is 
supposed to take place in a program. They are generally in the areas 
of "content" and "instructional design". Minor flaws are those which ' 
may slightly detract from the programs effectiveness. They are gen- 
erally in the area of. "technical design" or are weaknesses in meeting 
the needs of the specified users. One minor flaw by itself is hot 
necessarily cause for a^ lower ratirig. Sever&l minor flaws should 
result in. a supplementary status and more than 5 (arbitrary figure 
for the sake of establishing a line) should be cause for- rejection. 

Examples of major flaws: 

1. ..The sequence of the presentation is ineffective. The program may 

*'hop around" dealing with concepts in a disorganized fashion 

2. The sequence of the presentation may /completely miss an impor-tant 
concept which makes all further explanations confusing to the ' 

, student. v » ' 

3. The depth of .the content may be ineffective. Generally this will 
happen if the concepts are not g \ r**. sufficient depth of treat- 

' ment# e.g. one page is given when ..c. : " .'e necessary. As a 
result, the various topics are presanL.^o too rapidly arid the user • 
has too much material to assimilate at one time. ' ^ 
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4. The depth of the content does not include any--(or sufficient) 
practice. . 

5. There is insufficient student interaction in the program.. Generally 
the program is a "page turner" or it has limited student interaction 
even in the drill section. 

6* The program accepts incorrect answers or rejects correct answers. 

?• Feedback does not 'go beyond indicating if the answer is right or 
wrong. ,There is no explanation available as to why an answer 
may be wrong . t. , 

Examples of minor flaws: 

1. Material on the screen is cluttered, poorly displayed. 

2. Minor spelling or grammatical errors. > 

3. Insufficient user support materials. 

A. Student intei?action is not consistent between pushing return or 
pushing the space bar. 

5, Student is' not able to control the number of questions in a drill. 

6. Sound is distracting. 
Graphic displays are too slow. 

8. Definition of characters is hard for elementary students to read. 

9. Language is too difficult for the intended users. 
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V NECESSARY FEATURES OF TUTORIAITDRILL PROGRAFS 



The following should be considered as minimum requirements for tutorial 
and drill and practice programs. Except in exceptional cases, programs 
which do not have all seven features described below should be rejected. 

1. Orderly Structure 

Questions should be posed in an orderly progression so that the 
student can learn from the drill. This may be achieved by formatting 
the material into distinct units pr by leading the student through a 
^steadily more difficult series, of tasks. Drills which "hop around" 
or which have "quantum leaps" between the tasks should be rejected. 

2. Randomization 

A student should be able to redo the drill a number of times. 
Ideally, the questions should be different each time. If the nature 
of the material is such that this is impossible, the questions should \ 
be drawn from a bank of questions and should be presented in random 
order. • 

3. Appropriate Interaction 

In a tutorial, there should be some interaction between the program 
and the user. 

In a drill, a user will be inputting. answers but the type of inter- I 
action can limit a drill's effectiveness. . ^ 

a) Multiple choice questions can be ineffective if the user can- 
easily guess the answer without having to do the question. "For 
example, in the problem 10048 minus 2197, all alternatives should have 
a "1" in the last place. If only one answer does, it can be correctly 
selected without actually doing the question. 

b) Some interactions test the student's ability to type rather than 
knowledge of the material. . 

e.g. having to type in long, complicated words; timed questions where 
, words or long involved numbers have to be entered; allowing a 
user only one attempt at a question. 

A. immediate Feedback 

The student should be made aware immediately if he/she is right or 
wrong. Passing the student* on to the next question in a drill after , 
a correct answer has been entered is appropriate if the student is 
aware' that this is how correct answers are treaited. If an answer is 
wrong, there 'should be some immediate indication that the student's 
response was wrong. 
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Accurate Feedback 

Programs should accurately assess the student's input* This includes ■■ 
giving the correct answer as well as correctly evaluating a student's 
input. Ideally the program should also distinguish between an 
incorrect response and an incorrect format. ' 

Quantitative Feedback 

A program should give the student (and the teacher) the_results at the 
end of the drill. Ideally, a user should also know how many questions 
there are in the drill and how many have been done. 

Restructuring 

When an incorrect response or a sequence of incorrect responses has 
been entered, the student.' s- path through the program should be 
restructured in such a way that he/she learns why the answer was 
wrong. This is a critical element in instruction and unfortunately 
it is often missing in the courseware currently on the market. 
Something' more should happen than telling the student what the answer 
should have been. Examples of, restructuring are: 

a) Corrective feedback; hints and cues which direct the student on a 
second attempt " 

^ ' u i - ■ ■■ . ■ 

b) Availability of help pages y 

c) Remedial loops; a student may be directed to. a section of the 
program where further instruction is given (usually at a lower level) 
and/or where other drill questions lead the student through his/her 
weakness . > 

. ■ • - . ■ " I , ■ 

d) The program may loop back to a previous page and the student then 
continues from that point. This is not necessarily effective as the 
original explanation may be the reason for the confusion. 

e) The level of; difficulty mry be changed. This can be effective if 
the student is bifought back up to the desired level. 



f ) The number of questions may be increased or decreased depending 
upon the student's achievement. 



VI DEFINITIONS 



PRESCRIBED .LEARNING RESOURCES are those "learning resources approved by 
the Minister as being most appropriate for meeting the majority of goals 
and objectives for courses , or substantial components of courses , outlined 
in provincial Programs of Study", 

Evaluators do not have to make this recommendation- it is made by the 
Curriculum Committee. Evaluators just make the recommendation that the 
courseware is worth looking at by the Curriculum^ Committee. 

RECOMMENDED LEARNING RECOURCES are "those learning resources approved by 
Alberta Education because they complement Prescribed Learning Resources . 
by making an important contribution to the attainment of one or more of 
the major goals, of courses outlined in the provincial Programs of Study". 

As with prescribed materials, evaluators only recommend that specific 
courseware be examined further. This examination may result in a prescrib 
status, a recommended status or even a supplementary status. 

SUPPLEMENTARY LEARN I NG ^ RESOURCES are "those additional . learning resources 
identified by teachers, school boards or Alberta Education to support 
courses outlined in the provincial Programs of Study by .reinforcing or 
enriching the learning experience". 

Teachers may identify supplementary resources but this catejgory should 
not/be a dumping ground for all courseware that is less than "Recommended-' 
Courseware in this category should still be effective, it should meet the 
needs of- some target audience (perhaps not the target audience specified) 
and it should support some' of the goals of the curriculum. The range of 
potential use of the courseware may be limited, but What it does cover, 
it covers effectively. 



VII GLu^SAKY OF TERMS 



In the fields of microcomputers and education, terminology can easily be 
misinterpreted. So that . everyone using the Guide will interpret words in 
the same way, a brief glossary is provided below. Each definition conforms, 
to some standard use of the term but is hot necessarily a composite of ail 
uses nor the most common use. 

Content Facts, terms, ideas, concepts, principles , theories and 

contructs which make uj) the subject matter of an instructional 
package. 

Courseware Software and printed materials which support instruction in 

a complete course of study or a definite subset of a course. 
In this sense, it is not required that all instructional 
activities be supported by the package or that the tutorial 
mode of instruction be employed. Even a 30 minute student 
activity aimed at one objective can be courseware in this 
definition. 

Documentation Written material for the teacher to use with the computer 
program (also called a support booklet). 



DOS 



Drill & 
Practice Mode 



Games 



Graphics 



Hardware 



"Disk Operatilig System", an operating system which includes 
the capability . of controlling and coordinating the functions 
of a disk drive within 'a computer system. 

A type of computer program which acts as driilmaster by 
providing repetitive practice. on a skill or set of facts. 
The mode assumes that instruction in the concept skill pr 
process has already been given. Ideally a drill and 
practice program should : 1) provide random- selection of 
exercises; .2) record student progress; 3) tailor instruct-- 
ions, exercises and feedback to the individual; and 4) offer 
corrective feedback upon the input of incorrect responses. 

Programs in the instructional gaming mode call upon the user* 
to apply one or mbre specif ic skills or concepts within a 
";game" , environment . The game dimension, of the environment 
includes the, conditions of competition with self or others^ 
specific (if arbitrary) rules, the need to develop a winning - 
strategy and the^ introduction of random events to force 
revision of strategies. The instructional dimension of the 
environment demands that the development both of identifiable 
subject matter skills or concepts and of general problem 
solving skills he a goal of the exercise. 

Images displayed on a video screen or printer which are ]' 
generated by a -computer program. V 

Equipment, including computers; disk drives, cassette player.s, 
cables and monitors. 



Materials 



Microcomputer 



Package 



Problem Solver 



Program 



Simulation 



■/ 

Software 



Tutorial 



Books, folders, envelopes, worksheets and similar items. 
"User Support Materials" are items vjhich support the 
activity of .a person using a computer program. • , 

A computer system, including peripheral hardware; such 
as disk drive and monitor, based on a microprocessor 
(or "chip") and having a typewii-iter-like keyboard. 

One or more computer programs with related materials and 
the storage medium. 

A type of computer program which proc<>sses data for a 
student defined problem. 

A computer program, written in Basic, Pascal, machine 
code or other computer programming language. The routine^ 
and operations which instruct the computer. 

Programs in the simulation mode attempt to represent key 
aspects of some environment within which the user will , 
experience the neces'sity of making decisions and will be 
informed of the results of his decisions without experi- 
encing the real consequences of possible misjudgements. 
The time required to develop and use simulations is 
justified in situations where actual experience is ruled 
out due to extreme expense, safety consideration or the 
time required for the actual experience. Simulations 
include problem solving tasks (e.g. diagnosis of illnesses, 
genetic experiments, testing theoretical models)'* 
procedural tasks (e.g. acid titration, blasting, breeding 
of organisms) , or performances (e.g. control of water 
pollution). 

Computer programs. Including courseware, operating systems 
and languages. . 

iFor programs in this mode, the computer assumes the total 
responsibility for instruction. Tutorials are characterized 
by a dialogue between the student and the computer in which 
the direction and the level ^of the dialogue are shaped by ^ 
the student's input. Ideally, tutorials should tailor and 
present instruction to the individual, introduce new - 
concepts, provide exercises and tests, monitor student 
responses, give immediate. reinforcement and corrective - 
feedback, and keep records. 



Some definitions in this glossary have been taken from Evaluator's Guide 
published by MicroSIfT and The Computing Teacher.. ^ 



